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Abstract 

We compare the "unified approach" for the estimation of upper limits with 
an approach based on the Bayes theory, in the special case that no events are 
observed. The "unified approach" predicts, in this case, an upper limit that 
decreases with the increase in the expected level of background. This seems 
absurd. On the other hand, the Bayesian approach leads to a result which is 
background independent. An explanation of the Bayesian result is presented, 
together with suggested reasons for the paradoxical result of the "unified ap- 
proach". 

1. INTRODUCTION 

The study of a new phenomenon in science often ends up in a null result. However it might be of great 
importance to set upper limits, as this will help our understanding by eliminating some of the theories 
proposed. 

The determination of upper limits is presently a hotly debated issue in several fields of physics. 
Many papers have been devoted to this problem and different solutions have been proposed. In particular 
the problem has been discussed in paper [jl|] ("unified approach") and, more recently, in papers [Q, |p, 
based on the Bayes' theory. The use of the "unified approach" (FC) to set upper limits or confidence 
intervals is recommended by the PDG [EJ]. The "unified" and the Bayesian approaches are very different, 
not only in the sense that they lead to different numerical results but more radically in the meaning 
they attribute to the quantities involved. These differences lead to intrinsic problems in any comparison 
of their separate results. The purpose of this letter is to try to throw some light on this contentious and 
important issue. We shall show that the Bayesian approach is the correct one. If our argument is accepted 
by the scientific community, many debates about upper limits will be clarified. 

2. THE BACKGROUND DEPENDENCE PUZZLE 

According to the (FC) "unified approach" the upper limit is calculated using a revised version of the 
classical Neyman construction for confidence intervals. This approach is usually referred to as the "uni- 
fied approach to the classical statistical analysis", and it aims to unify the treatment of upper limits and 
confidence intervals. On the Bayes side, according to ||], the upper limit may be calculated using 
a function 1Z that is proportional to the likelihood. This function is called the "relative belief updating 
ratio" and has already been used to analyse data in papers []|, The procedure has been extensively 
described by G. D' Agostini in [0]. 

Comparison between the two approaches is difficult for the general case. But we have noticed a 
special case which is easier to discuss. In this case the greater efficacy of one approach compared to 
the other one seems clear. This case is when the experiment gave no events, even in the presence of a 
background greater than zero. 

When there are zero counts, the predictions obtained with the two methods are different and both 
are -intuitively- quite disturbing. Our intuition would, in fact, be satisfied by an upper limit that increases 
with the background level, and this is, in general, the case when the observation gives a number of events 



of the order of the background. However, when zero events are observed, the "unified approach" upper 
limit decreases if the background increases (a noisier experiment puts a better upper limit than a less noisy 
one, which seems absurd) while the Bayesian approach leads to the predictions that a constant upper limit 
will be found (the upper limit does not depend on the noise of the experiment). Various papers[||, ||, 10] 



have been devoted to the problem of solving some intrinsic difficulties with the "unified" approach: 
in particular to solving the problem of "enhancing the physical significance of frequentist confidence 
intervals" , or to imposing "stronger classical confidence limits" [|h. In this latter article the proposed 
method "gives limits that do not depend on background in the case of no observed events" (that is the 
Bayesian result !). 

In what follows we will give an explanation for the two results. 

We remind the reader that the physical quantity for which a limit must be found is the events rate 
(i.e. a gravitational wave burst rate) r. Here we will assume stationary working conditions. For a given 
hypothesis r, the number of events which can be observed in the observation time T is described by a 
Poisson process which has an intensity equal to the sum of that due to background and that due to signal. 

In general, the main ingredients in our problem are that: 

• we are practically sure about the expected rate of background events r b = n b jT but not about the 
number of events that will actually be observed (which will depend on the Poissonian statistics). 
T is the observation time; 

• we have observed a number n c of events but, obviously, we do not know how many of these events 
have to be attributed to background and how many (if any) to true signals. 

Under the stated assumptions, the likelihood is 

e -(^)T ((r + rfe)T) n c 

f(n c \r,r b ) = - , (1) 

n c ! 

We will now concentrate on the solution given by the Bayesian approach. 
The "relative belief updating ratio" 1Z is defined as: 

This function is proportional to the likelihood and it allows us to infer the probability that rT 
signals will be observed for given priors (using the Bayes's theorem). 

Under the hypothesis r b > if n c > 0, 1Z becomes 



K(r;n c ,r b ,T) = e ~ T 1 (l + -J . (3) 

The upper limit, or -more properly- "standard sensitivity bound" [j7j], can then be calculated using 
the 1Z function: it is the value r ssb obtained when 

H{r ssb ;n c ;r b ;T) =0.05 (4) 

We remark that 5% does not represent a probability, but is a useful way to put a limit independently 
of the priors. 

Eq. H when no events are observed, that is, when n c =0, becomes: 

K(r) = e~ rT (5) 



Thus putting n c = in Eq. ^ we find r ssb = 2.99, independently of the value of the background 

n b . 



We will not describe the well known (FC) procedure here, but we would just observe that, accord- 
ing to this procedure, for n c = and n b = 0, the upper limit is 3.09 (numerically almost identical to the 
Bayes' one) but it decreases as n& increases (e.g. for n c = and n& = 15 the upper (FC) limit at 95% 
CL is 1.47). 

In an attempt to understand such different behaviour we will now discuss some particular cases. 
Suppose we have n c = and rib ^ 0. This certainly means that the number of accidentals, whose 
average value can be determined with any desired accuracy, has undergone a fluctuation. The larger the 
rib values, the smaller is the a priori probability that such fluctuations will occur. Thus one could reason 
that it is less likely that a number n gw of real signals could have been associated with a large value of rib, 
since the observation gave n c = 0. 

According to the Bayesian approach, instead, one cannot ignore the fact that the observation n c = 
has already being made at the time the estimation of the upper limit comes to be calculated. The 
Bayesian approach requires that, given n c = and rib 7^ 0, one evaluates the chance that a number n gw 
of signals exists. This chance of a possible signal is applied to the observation that has already been 
made. 

Suppose that we have estimated the average background with a high degree of accuracy, for ex- 
ample nj,=10. In the absence of signals, the a priori probability of observing zero events, due just to a 
background fluctuation, is given by 

in = f(n c = 0\n b = 10) = e~ nb = 4.5 • 10~ 5 (6) 

Now, suppose that we have measured zero events, that is n c =0. In general n c = (n^ + n gw ). It is 
now nonsense to ask what the probability that n c =0 is, since the experiment has already been made and 
the probability is 1. 

We may ask how the a priori probability would be changed if n gw signals were added to the 
background. We get 

fsn = f(n c = 0\n b = 10, n gw ) = e ~K+^) (7) 

It is obvious that f sn can only decrease relative to f n , since we are considering models in which 
signal events can only add to noise events[]. 

The right answer is guaranteed if the question is well posed. Given all the previous comments, 
the most obvious question at this point is: what is that signal n gw which would have reduced the 
probability f n by a constant factor, for example 0.05 ? 



f sn = f n • 0.05 = e- n " ■ e~ n ^ (8) 
Using Eqs. ||, [7] and |] the solution is: 

e -n gw = o.o5 (9) 



that is: 

n gw = 2.99 (10) 

1 In a gravitational wave experiment signals may add up to the noise with the same phase, thus increasing the energy of the 
combined effect, or with a phase opposite to that of the noise, thus reducing the energy. They can in particular add up also to 
noise events, even if we expect this to happen with a very low probability, as we know that the events due to the signal are very 
"rare" compared to the events due to the noise. 

Anyway, in principle, the presence of this fact will lead to the prediction of a signal rate that increases with the background: 
in fact the probability that one background event be cancelled by a signal event increases, as rif, increases. Thus, if we, at least 
in part, attribute the observation of n c =0 to a cancellation of background events due to the signal the final limit on r should 
increase. 

In the modelling we usually, as reasonable, consider this effect be negligible. If this is not the case then it must be properly 
modelled in the likelihood. 



Now suppose another situation, nb=20, thus /„ = 2.1-10 9 . Repeating the previous reasoning 
we still get the limit 2.99. 

The meaning of the Bayesian result is now clear: we do not care about the absolute value of the 
a priori probability of getting n c = in the presence of noise alone. The observation of n c = means 
that the background gave zero counts by chance. Even if the a priori probability is very small, its value 
has no meaning once it has happened. The fact that the single background measurement turned out to 
be zero, either due to a zero average background or due to the observation of a low (a priori) probability 
event, must not change our prediction concerning possible signals. 

For n c = we are certain that the number of events due to the background is zero. Clearly this 
particular situation gives more information about the possible signals. In the case n c ^ 0, instead, it is 
not possible to distinguish between background and signal. The mathematical aspect of this is that the 
Poisson formula when n c = reduces to the exponential term only, and thus it is possible to separate the 
two contributions, of the signal (unknown) and of the noise (known). 

We note that the different behaviour of the limit in the unified approach is due to the non-Bayesian 
character of the reasoning. In such an approach an event that has already occurred is considered "im- 
probable": given the observation of n c = they still consider that the probability 

fsn = /K = 0\n b ,n gw ) = e -(**+"*«) (11) 

decreases as increases. As a consequence they deduce that to a larger n& corresponds a smaller upper 
limit n gw . 

Given the previous considerations, we must now admit that our intuition to expect an upper limit 
that increases with increasing background, even when n c = 0, was wrong. We should have expected to 
predict a constant signal rate, as a consequence of the observation of zero events, independently of the 
background level. 



3. CONCLUSION 

We have compared the upper limits obtained with the (FC) "unified" and with the Bayesian procedures, 
in the case of zero observed events. 

We believe that the greater efficacy of the Bayesian approach compared to the (FC) method, 
demonstrated for the case n c = 0, is a strong indication that the Bayesian method -natural, simple 
and intuitive- is the correct one. Thus we agree with the proposal in[{7|] that this method should be 
adopted by the scientific community for upper limit calculations (see, for example, [|ll|] on upper limits 
in gravitational wave experiments). 
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